Drive assist systems called around view monitors, wrap-around 3D monitors, etc. are known. This system uses a plurality of in-vehicle cameras installed in a vehicle to pick up images and synthesizes those images so as to display an image around the vehicle.
In this system, in-vehicle cameras respectively installed at the front, rear, right and left parts of a vehicle first pick up four images, a front camera image 1F, a rear camera image 1B, a left camera image 1L, and a right camera image 1R, respectively, as illustrated in FIG. 1. Then the drive assist system synthesizes these picked-up images 1 so as to generate and display an image that looks as if it were a result of taking the surroundings of the vehicle looking down from above, an example of which is illustrated as a looking-down image 2 in FIG. 2. In looking-down image 2, front camera image region 2F, rear camera image region 2B, left camera image region 2L, and right camera image region 2R are results of deforming partial regions in front camera image 1F, rear camera image 1B, left camera image 1L, and right camera image 1R, respectively.
In this system, when each camera independently performs control such as Automatic Gain Control (AGC) or the like, differences may be caused in brightness between the picked-up images 1 obtained by the respective cameras, sometimes resulting in a situation wherein differences in brightness cause discontinuous portions at the boundaries between the respective regions over the picked-up images 1 in the synthesized image. In the example illustrated in FIG. 2, discontinuous portions may occur at the boundaries, as depicted by the thick lines in the looking-down image 2, between front camera image region 2F, rear camera image region 2B, left camera image region 2L, and right camera image region 2R. Differences in brightness between the picked-up images 1 are eliminated by for example correcting the brightnesses of the picked-up images 1 obtained by the respective cameras on the basis of pixel values in the image regions in overlapping portions of viewing fields between the cameras. There are some known techniques for this purpose.
For example, as the first technique, a technique is known that generates a brightness-corrected image by using a correction coefficient calculated on the basis of pixel values in image regions of overlapping portions of input images, and generates a synthesized image from that corrected image so as to reduce discontinuity at boundaries. Also, for example, as the second technique, a technique is known that first performs viewpoint transformation on an input image for each camera, estimates correction coefficients, performs correction of overlapping regions in the images that received the viewpoint transformation, and synthesizes the corrected images (see Patent Documents 1 through 4 for example).
As a different background art, a technique is known that synthesizes camera images obtained by a plurality of cameras installed around a vehicle so as to generate a wrap-around image in which the virtual viewpoint can be changed arbitrarily (Patent Document 5 for example).
Patent Document 1: Japanese Laid-open Patent Publication No. 2002-27448
Patent Document 2: Japanese Laid-open Patent Publication No. 2006-173988
Patent Document 3: Japanese Laid-open Patent Publication No. 2008-28521
Patent Document 4: Japanese Laid-open Patent Publication No. 2010-113424
Patent Document 5: International Publication Pamphlet No. 2012/017560
According to the first and second techniques described above, correction is conducted on a picked-up image or a viewpoint-transformed image for each camera so as to complete the generating of corrected images, and thereafter the corrected images are synthesized. When this type of image processing is executed, an image memory for storing corrected images is used in addition to respective image memories for holding picked-up images or viewpoint-converted images and a synthesized image. Also, a capturing process of picked-up images or a viewpoint-converted image, a brightness correction process, and an image synthesization process that are executed in this image process are executed serially. A brightness correction process is a process executed on all pixels constituting picked-up images or a viewpoint-converted image, and an image synthesization process is started after the completion of a brightness correction process on all pixels. This elongates the latency between picking up of images by cameras and the displaying of the synthesized image when an image process as describe above is executed. Short latencies are preferable for drive assist apparatuses, for which driving safety is desired.
Also, as an example, a case is discussed in which a synthesized image is generated without conducting the brightness correction on images obtained by respective cameras and thereafter the brightness correction is conducted on the synthesized image. Because the synthesized image has not received brightness correction in this case, there are brightness differences at boundaries between picked-up images obtained from the cameras. In the image process in this case, calculation of correction values for eliminating brightness differences between pixels is performed for all pixels constituting the synthesized image and thereafter brightness correction is conducted on the synthesized image. Accordingly, when this type of an image process is conducted, an image memory is used for holding a synthesized image that has not received brightness correction. Also, it is not possible to perform calculation of correction values before the completion of the generating of the synthesized image in this image process and accordingly the latency between the obtainment of images by the cameras and the displaying of the synthesized image becomes longer. Also, when for example the viewpoint position has been changed in the synthesized image, the relationships between the positions, in the synthesized image, corresponding to boundaries between picked-up images obtained from the cameras also change, causing the need to recalculate correction values even when there are no brightness differences in the picked-up images obtained from the respective cameras.